Westudy theasynchronous stochastic gradient descent algorithm fordistributed training overn workers which have varying computation and communication frequencyovertime.

The analysis for Hogwild algorithm seems to be new as well, but I didn't go through the proof of that part. Significance: The new analysis supporting asynchronous update when features are sparse, will be an interesting message to the community. However as the assumption in Eq.(2) excludes many interesting problems, I don't think it will have a large impact. Q2: Please summarize your review in 1-2 sentences The paper suggests an analysis showing that the optimal bounds can be achieved by their suggested algorithm up to some factors, where the algorithm is claimed to have linear speedup with the number of cores and to benefit from the sparsity of input features. However, its applicability seems to be limited due to a rather restrictive assumption it is based upon, which makes the suggested method sensitive to the choices of stepsizes, and the other assumption on asynchronous update delays.

Spatial data are central to applications such as environmental monitoring and urban planning, but are often distributed across devices where privacy and communication constraints limit direct sharing. Federated modeling offers a practical solution that preserves data privacy while enabling global modeling across distributed data sources. For instance, environmental sensor networks are privacy-and bandwidth-constrained, motivating federated spatial modeling that shares only privacy-preserving summaries to produce timely, high-resolution pollution maps without centralizing raw data. However, existing federated modeling approaches either ignore spatial dependence or rely on synchronous updates that suffer from stragglers in heterogeneous environments. This work proposes an asynchronous federated modeling framework for spatial data based on low-rank Gaussian process approximations. The method employs block-wise optimization and introduces strategies for gradient correction, adaptive aggregation, and stabilized updates. We establish linear convergence with explicit dependence on staleness, a result of standalone theoretical significance. Moreover, numerical experiments demonstrate that the asynchronous algorithm achieves synchronous performance under balanced resource allocation and significantly outperforms it in heterogeneous settings, showcasing superior robustness and scalability. Keywords: Asynchronous federated learning, distributed spatial modeling, Gaussian processes, low-rank approximation, block-wise optimization.

We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchronous versions of these algorithms--a crucial requirement for modern large-scale applications--have not been studied.

Neural Information Processing Systems http://nips.cc/

We thank all the Reviewers for their time and raising several interesting questions. Please see our responses below. Reviewer #1: We will try to reduce dependence on the Supplement. The name V ol in 3.3 refers to V olume, which for the ellipsoid We will add this definition. Reviewer #2: We will add a comment comparing the convergence rate of LSGD to other distributed methods.